Meta-Analysis: The Librarian as a Member of an Interdisciplinary Research Team

META-ANALYSIS IS A quantitative statistical tool for combining research studies with a small study population to achieve a larger effect in size. It combines the talents of subject experts, statisticians, meta-analytic specialists, information management professionals, and librarians, creating a multidisciplinary team. This article will explore the interdisciplinary nature of interdisciplinary research, provide a brief explanation of the Integrative Review of Research (IRR) of which meta-analysis can be a part, and describe the librarian’s role or roles in the various stages of the project. Finally, a look at developing trends or issues in the area will be discussed. THEINTERDISCIPLINARY NATUREOF INTERDISCIPLINARY RESEARCH Julie Thompson Klein’s paper in this issue of Library Trendssets the stage for a discussion of the impact that interdisciplinary research has on researchers, the library and its staff, academic departments, and their parent institutions. Academic administrators are grappling with the reality of shrinking state funds and are applying pressure on researchers to be totally self-sustaining. Researchers seeking federal support for their projects are finding dwindling funding sources which means that the competition for grant support is more competitive than ever before. One way to gain an advantage is to submit a grant proposal that crosses disciplines. Klein describes the interdisciplinary approach evidenced in several broad disciplines. More specifically, the area of health sciences research is undergoing this same phenomenon. A perusal of the titles of projects Jack T. Smith, Jr., Lister Hill Library of the Health Sciences, The University of Alabama at Birmingham, 1700 University Blvd., Birmingham, AL 352940013 LIBRARY TRENDS, Vol. 45, No. 2, Fall 1996, pp. 265-79 01996 The Board of Trustees, University of Illinois 266 LIBRARY TRENDS/FAI,I, 1996 funded by the Department of Health and Human Services, Public Health Service, illustrates this fact. The following three exemplars were chosen from a list produced from a search of the CRISP database as mounted on the World Wide Web (gopher://gopher.nih.gov) site at the National Institutes of Health. “CNS Effects of Alcohol-Cellular Neurobiology” has a stated purpose “to continue its long term, cooperative, interdisciplinary research.’’ One of its subthemes is “the molecular and cellular mechanisms of short term ethanol intoxication, and its endocrine, metabolic and behavioral concomitants” which demonstrates the variety of disciplines that are involved. Christine Cassel was awarded a Geriatric Leadership Academic Award. This award “will assist her in expanding interdisciplinary research in aging at the University of Chicago and in the broader academic community throughout this city.” Cassel will deal with a new basic science research facility,a new Department of Health study, as well as foster “collaborative agmg research in the social and biomedical sciences.” A prime example of one researcher blending several disciplines into a research project. Finally, the Western Consortium for Public Health submitted a project entitled “Meta-Analysis-Social Relationships and Drinking Outcome.” The consortium “proposed to determine the association between social relationship factor and alcoholism treatment drinking outcomes.” Among the areas that the study will address are alcoholism, alcoholism therapy, psychosocial rehabilitation, and quality of life. If researchers are to conduct interdisciplinary projects, they must have appropriate outlets for dissemination. Evidence of opportunities for publication of multidisciplinary research in the health sciences may be gleaned from the List of Serials Indexed for Online Users (National Library of Medicine, 1996). Such titles as Cardiovascular and Interuentional Radiology, Health and Social Work, and Social Science and Medicine demonstrate the kind of breadth to be found. In Medical Subject Headings, the National Library of Medicine (NLM) (1989) defines meta-analysis as “a quantitative method of combining the results of independent studies (usually drawn from the published literature) and synthesizing summaries and conclusions which may be used to evaluate therapeutic effectiveness, plan new studies, etc., with application chiefly in the areas of research and medicine” (pp. 1-40). Meta-analysis began to be used as an index term that year. However, Gene V. Glass (1976) had begun using the term in 1976 (p. 3 ) . The use of statistical techniques to combine research results might go back to Legendre in 1805 and his development of least squares (Cook et al., 1992, p. 6). In combining the results of individual agricultural experiments, two differSMITH/ META-ANALYSIS 267 ent approaches were taken. One tested for statistical significance of the combined results. The other relied “on estimating treatment effects across studies” (Hedges & Olkin, 1985, p. 1). By the 1930s, meta-analysis research began to appear in the social sciences (Cook et al., 1992, p. 6). It was not until the late 1970s that meta-analysis began to appear in the medical literature (Schell & Rathe, 1992, p. 219). After Glass’s 1976 article, other researchers also began to refine metaanalysis procedures and publish their results (Hedges & Olkin, 1985; Rosenthal, 1984; Wolf, 1986; Cooper, 1979). The medical community was still hesitant to accept meta-analysis studies and to believe in the validity of the results. A group at Oxford University in the 1980s began to change this. This group “took the approach of gathering all studies, published and unpublished, and excluding those that used different endpoints” (Schell &Rathe, 1992, p. 219). Using studies on therapeutic issues, the British called their research “overviews” and they recommended that their conclusions be used in clinical trials for a further check of the validity of their results (Yusuf, Collins, et al., 1985; Yusuf, Petro, et al., 1985). INTEGRATIVE RE I WOF RESEARCH Cooper introduced procedures for doing an Integrative Review of Research in 1982. He described it as the application of the research process to a collection of studies (Cooper, 1982). To achieve reliable results, there must be rigorous adherence to the rules of scientific inquiry with special attention to threats to validity. Meta-analysis is contained within the scope of an Integrative Review of Research (Smith, 1991, p. 48). Cooper (1984) detailed a five-stage process for conducting an IRR. These stages are: (1) problem formulation, (2) data collection, (3) data evaluation, (4) analysis and interpretation, and (5) public presentation (Cooper cited in Smith et al., 1991, p. 48). The librarian or information professional can be involved in any or all stages. THEROLEOF THE LIBRARIAN The literature on what the role of the librarian should be in a metaanalysis project is scant. The Committee on National Statistics (CNSTAT) of the National Research Council held a workshop on the future of metaanalysis in 1986 (Wachter & Straf, 1990). The goal “was to assess the role actually played by meta-analytic methodologies in current practice” (p. xiii), identify strengths and limitations, and suggest priorities for future research. The major players in meta-analysis, Harris Cooper, Robert Rosenthal, Larry Hedges, and Ingram Olkin, made presentations at this workshop. For the purposes of this article, the relevant sections are those concerning the literature review. One group actually performed a metaanalysis on treatment of aphasia specifically for this workshop (Wachter & Straf, 1990, pp. 29-46). In their concluding paragraph to the section 268 LIBRARY TRENDS/FALL 1996 on researching the literature, they acknowledge that this retrieval process is “the most critical phase in a quantitative literature review” (Wachter & Straf, 1990, p. 34). Their next sentence provides an insight into how the entire meta-analysis process is perceived by the subject expert. “It became obvious that this phase [literature review] could be completed only by a subject matter specialist, that is, a person knowledgeable about aphasia, in our case Dr. Fromm. She felt, however, that a measure of reliability would have been added to the project by having two subject experts, rather than one, involved in this stage” (Wachter & Straf, 1990, p. 34). In her reaction to this study, Nan Laird recognized the importance and the enormity of the task for the meta-analysis researcher in undertaking a comprehensive literature review (Wachter & Straf, 1990, p. 48). All contributors to this monograph agreed with Harris Cooper’s statement: “Nobody’s out there searching forever. If they are, they’re certainly not the folks who are here. They’re still at the library and have never published” (Wachter & Straf, 1990, p. 169). Cooper went on to say that those who are conducting a search of the literature have an obligation to strive for comprehensiveness. That if researchers are going to say to a reader that they have examined the literature and can describe and summarize it, then the researcher should be cognizant of what is going, and has been going, on. Cooper also firmly believes that it is vital that the researcher be “incredibly explicit” about the process they use to retrieve the literature and the criteria used to determine what to cite (Wachter & Straf, 1990, pp. 168-69). What is noticeably absent in the monograph is the mention of any role for a librarian to play. A reading of the monograph suggests that the contributors performed their own searches, selected the databases or print indexes to be searched, and formulated their own strategies. An unstated question is, Would it have been beneficial if a librarian had been consulted regarding the search? The librarian would need to have experience in conducting an in-depth reference interview, training in online search techniques, and possess the requisite skill for online retrieval of literature. At the very least, the librarian’s search results could be checked against the researcher’s result for consistency. Schell and Rathe (1992) provide a brief historical perspective on metaanalysis and its use in medicine. The librarian is seen as having a role in defining the inclusion parameters, data collection and bibliography, and summary of findings. The authors believe that a librarian who is knowledgeable in meta-analysis techniques will be able to choose the degree of involvement in a project. Because of the librarian’s training and experience in citation analysis and literature searching, they should play an important role in this type of research (Schell & Rathe, 1992, p. 221). Smith, Smith, and Stullenbarger (1991) used Cooper’s (1991) fivestage process for conducting an integrative review of research to present a detailed explanation of the process so that librarians would have a betSMITH/META-ANALYSIS 269 ter understanding of the needs of the researcher and how these needs could best be served (Smith et al., 1991, pp. 47-72). As part of the presentation, they formulated flow charts to illustrate various decision points in the process. In each of Cooper’s five stages, Smith et al. discussed the specific needs the research members would have and how the librarian could address and meet those needs. Smith, Smith, Stullenbarger, and Foote (1994) followed up this theoretical article by taking an actual topic on head-injured adults and presented a practical application of the process (pp. 57-72). As in the earlier article, the specifics of the librarian’s role was discussed and illustrated by describing exactly what assistance should be given to the research team and at what point. Mead and Richards (1995) describe the process used by The Center for the Evaluative Clinical Sciences (CECS) of Dartmouth Medical School. A typical meta-analysis research team is interdisciplinary and would be comprised of a team leader, statisticians, subject specialists, and grant source experts. The Technology Assessment Program is a unit within CECS with a librarian as an integral participant in their projects. The librarian is viewed as a valuable resource in the “selection of bibliographic databases and vendors, planning and testing of search strategies, and use of Medical Subject Heading (MESH) tools and other controlled vocabularies’’ (Mead & Richards, 1995, p. 462). Other areas in which the librarian provides assistance include suggesting alternatives for finding references, focusing discussions among team members, reviewing search results, and providing regular updates. One of the more intriguing aspects Mead and Richards describe is the use of FileMaker-Pro for postprocessing of search results and networking the resulting database for the researchers to use in article selection. A template was designed so that the researchers can view the citations with the appropriate fields masked (i.e., authors and their affiliations). In reviewing the citations, all a researcher has to do is click on a button or box to indicate whether or not the article should be retrieved for further study. According to Mead and Richards (1995) : The benefits of this approach are many: It is not necessary to print thousands of references. The method is easy to use and support. The approach facilitates the recording of exclusion codes. The method simplifies and consolidates work effort. A single copy of the file can be networked. Articles not relevant are eliminated based on downloaded data instead of a hard copy, saving lots of trees and fees. (p. 463) Mead and Richards (1995) see an expanded role for the librarian in this process. At a time when the role of the medical librarian is in transition, being a valued member in the meta-analysis process can assist in defining a new role. As an active participant, the librarian can enhance 270 LIBRARY TRENDS/FALL 1996 and refine the literature analysis process and bring unique skills to the task of improving health care (p. 463). At the University of Alabama at Birmingham, an interdisciplinary meta-analysis research team has been in place since the late 1980s. The team consists of two meta-analysis experts, a statistician, and a health sciences librarian. While working mostly with researchers in the health sciences, the group is available to anyone on campus who wishes to investigate the possibility of conducting an integrative review of research. Subject areas which have been reviewed include oncology nursing research; nursing research of the adult head injured, cited previously; and quality of life of recovering cardiovascular patients. The group is currently working with the Alabama Board of Nursing evaluating the literature concerning continuing education. New territory is being explored in this project. The research team is applying existing integrative review of research processes to an entirely different arena-appellate case law. The team is composed of nursing education experts, meta-analysis methodologists, a statistician, and legal experts including an online research analyst from the Alabama Supreme Court Library who is also a lawyer. The methodology of this research as well as its findings will be published at the conclusion of the project. This “research in progress” is mentioned because there are two information professionals who are integral members of the research team. Both have been involved from the beginning of the project, but their roles and contributions are very different. The team drew on the Supreme Court librarian’s expert knowledge of the court system, the law, its terminology, and electronic legal databases. Because the group was going into uncharted water, the librarian acts as a sounding board and relevance check for ideas on procedure. The health sciences librarian has contributed his skills in identifying the appropriate nonlegal databases which contain information on how continuing education is discussed and applied to regulated or licensed occupations or professions. He has also had the opportunity to participate in coding the legal cases which were selected for inclusion in this study. The other team members value these contributions and view the librarian as having an integral role in meta-analysis research. An examination of Cooper’s (1984, p. 12) five stages demonstrates the role that the librarian can play in an integrative review of research or metaanalysis project. Cooper’s first stage is problem formulation. It is in this stage that the actual question(s) to be researched is (are) constructed. Usually, the question h a to do with a topic that has been discussed in the literature and that has had conflicting results associated with it. For example, a medical researcher might want to cull the literature to investigate the effect that a particular drug has on a certain disease. Asocial worker could investigate what effect, if any, early intervention had on spousal abuse. For a true SMITH/META-ANALYSIS 271 multidisciplinary topic, any quality of life study could bring together subject experts on medicine, nursing, psychology, physical therapy, occupational therapy, social work, education, or the humanities. Librarian involvement in this stage can take many forms. Naturally, it would be most helpful if the researcher would state the topic of interest and the fact that a meta-analysis on the topic is being considered. The librarian would then be presented with the opportunity to ask, during the reference interview, specific questions like What time period is needed? Are there specific aspects of a topic that should be present in each study? Is there a language restriction? Is there a country limitation as to publication or population? If the librarian is not familiar with the topic or a particular aspect of the topic, then a student-teacher atmosphere is needed with the librarian as student and the researcher as teacher. It is vital that the librarian gain an extraordinarily clear picture of what kind of information the researcher is requesting and at what breadth and depth and level. A more realistic scenario is one in which the researcher comes to the library and asks for assistance in finding information on a certain topic. With this typical request, even a standard reference interview might not elicit the fact that the patron wishes to embark on an integrative review of research or a meta-analysis on that topic. Remember at this stage the researcher is doing a preliminary investigation of the literature to see if an in-depth review would be worthwhile. Perhaps the only clue the librarian would receive is if the patron asks for reviews, meta-analytic studies, integrative reviews, research synthesis, or other combinations of these words. The use of these phrases by the patron to describe the type of articles that would be useful alerts the librarian to the possibility that the patron might be considering a meta-analysis project. It might be prudent at this point to inquire if the patron is beginning an integrative literature review or meta-analysis. As a result, both patron and librarian will be on equal footing in their information quest. Once the topic and its parameters have been discussed and agreed on, the next step is the selection of databases and formulating a search strategy that will elicit the desired results. At this point, the researcher does not need a thorough literature review. However, it is at this point that a decision is made by the researcher to continue or discontinue the project. For this reason, the researcher may augment the online search results with an issue-by-issue search of two or three top journals in the field. Granted, the researcher probably knows which titles these are, but the librariarz may be asked if a standard list ofjournals in this area exists. This list would serve as a verification that the appropriate journals have been included in this manual search. The results of this search process serve to alert the researcher, as well as the librarian, to missed articles and to verify the validity of the online search. It is quite possible that the researcher has been keeping a reprint file on the topic and this file may 272 LIBRARY TRENDS/FALL 1996 be culled for appropriate articles. All of these information sources serve to answer the question, “Should this project go forward?” A number of other decisions must be made. One is, which metaanalysis approach to use for the project. The ones most frequently used are Cooper, Hedges and Olkin, Rosenthal, and Hunter and Schmidt. Briefly, the “Cooper and Hedges and Olkin approach asks if there is a difference between experimental and control groups as to effectiveness of treatment outcome. Rosenthal ...asks if there is a relationship between treatment and outcome. Hunter and Schmidt account for artifacts, such as sampling errors, in primary studies and use Rosenthal’s approach” (Smith et al., 1994, p. 61). This decision is made in conjunction with formulating the “problem question.” It is in researching the answer to the problem question that the project is driven. The question must be carefully crafted so that it can be matched with the proper approach and in order for the results to be meaningful. At this point, the researcher evaluates what resources are on hand. Are the appropriate subject/content experts available and willing to participate in the project? Are methodology experts knowledgeable in metaanalysis or integrative review of research available and willing? Is a librarian who has the proper information skills, training, and experience on board? Is a statistician present to run the appropriate statistical tests? If the team members have committed their time, and the proper expertise is represented on the team, the last question is one of funding. Will the project have the proper financial support to see it through to its conclusion? If all of these questions are answered in the affirmative, then the project may proceed. Once the problem question is formulated and the proper resources are in place, then the team begins a discussion of the variables. Many of these variables are always included, such as date of publication or language, and are considered standard variables. Substantive variables are those items that are unique to the area being researched. For a study involving family violence, these variables might include family size, age of family members, and sex of family members. If teenage pregnancy is being studied, age of mother, the composition of the family unit, marital status of the mother, and age of the baby’s father might be substantive variables. When all the variables are identified, a coding form and a glossary are created. The coding form fits all of the variables, both standard and substantive, with a blank by each variable. The person filling in the coding form, the coder, reads each article or study and fills in the form appropriately. The variable may have a yes or no answer (e.g., is the mother married?), or it may have a list from which the coder chooses the appropriate answer (e.g., age at onset of condition 15-20, 21-25, 26-40). The glossary contains a definition of each term on the coding form. These SMITH/META-ANALYSIS 273 definitions should be decided upon by the team, and each definition must have a source from which the definition of the term was derived. The coder uses the glossary in conjunction with the coding form and its corresponding instructions to answer any questions about how any term or variable is being used or defined. There should be no doubt or ambiguity in the coder’s mind concerning any variable on the coding form. The ideal situation is that a person with no knowledge or background of the subject area being investigated should be able to code accurately. In fact, many subject experts and methodologists are not allowed to be coders because they might bring a hidden bias to that part of the project which would adversely affect the validity of the results. The librarian can play a role in this part by acting as an objective observer. Both the researchers as subject experts and the methodologists have unique points of view. The librarian can act as facilitator, keeping the discussion on-point and bringing an unbiased mind to the discussion. There is also a traditional role here-identifjmg dictionaries that the team might need in order to prepare the glossary. In addition to standard English language dictionaries, specialized subject dictionaries exist in many fields such as law, demography, ecology, and psychotherapy. It is important to the integrity of the entire project that the researchers be able to document a source for each piece of information, and the librarian can assist in identifymg and obtaining these sources. Cooper’s second stage, data collection, is perhaps the most intensive for the librarian. It is during this stage that comprehensive searches are performed to retrieve as many relevant study articles and sources as possible. In many respects, the roles and actions of the researcher and the librarian discussed in stage one are repeated but with a clearer focus and at a comprehensive level. In 1985, Cooper did a survey to discover how investigators performing meta-analytic studies found the studies they used for their research (Cooper, 1985). The three most frequently used sources were: (1)“references in review papers written by others”; (2) “references in books written by others”; and (3) “communication with people who typically share information with you” (Cooper, 1985, p. 1268). An online search of the literature was fifth. Cooper does state that online searches “appear to be making significant inroads on reviewing practices, and reviewers who use computer searches find them extremely useful” (p. 1268). Current meta-analysis studies present the results of the literature review. However, the details of the review consisting of what was searched, database selection, and search strategies are rarely included in the report. The researcher must keep a record of all online searches performed and their strategies, any manual searches that were done, whether in reprint files, files of colleagues, or issue-by-issue searches of personal or library journal collections, or any other searching techniques or sources that were used to identify studies for the project. This record serves two 274 LIBRARY TRENDS/FALL 1996 important functions. First, if a reader wants more detailed information about the search and retrieval process, the researcher has that record at hand and can provide the information. The second function is for replication. If a future researcher wishes to replicate the study or perform a similar study, this record serves as a baseline from which to begin. On a practical level, the librarian and the researcher examine the problem question and the variables identified for inclusion in the study. This information is used to review the initial search strategies and databases searched. As a result, the initial search parameters may be expanded or altered. To be as comprehensive as possible, many different publication types are now searched for possible studies. In addition to studies reported in the regular journal literature, basic literature reviews, conference proceedings, unpublished studies, government publications, monographs, and electronic journals and tables-of-contents may be searched (Smith et al., 1994, p. 63). Barbara Quint’s two-part series “Inside a Searcher’s Mind: The Seven Stages of an Online Search,” is an excellent and exhaustive presentation of the online search process (Quint, 1991a, 1991b). At the completion of the project, most, if not all, of Quint’s suggestions and ideas will have found a place in an online search of the literature. The librarian’s strengths are the knowledge of, and experience in dealing with, database construction, indexing practices, timeliness of the information added, update intervals, use of controlled vocabulary, full-text versus bibliographic databases, and how to refine and revise strategies once online. These strengths will be needed in dealing with the newest source of information, the World Wide Web. Librarians are playing a key role in organizing this information. The researcher will need to be aware of this resource, both its richness and its limitations. The same rules apply to these “publications” as to any print publication. Does the study meet the inclusion criteria? Is the methodology sound? Has it been reviewed? Ferreting out information from the Web requires a knowledge of how the various search engines that search and index Web sites are designed. Yahoo, AltaVista, Lycos, and Webcrawler are just some of the search engines available. However new and exciting these Web search engines are, the librarian and researcher are still dealing with a database. It is a different kind of database, but the questions about its informational content are not different. They are just avariant of the information needed about bibliographic or full-text databases. Questions like What sites have been indexed? How are the results organized? and What are the search and retrieval options? need to be answered if this resource is to be used effectively. Data evaluation is stage three of the meta-analysis process. Usually the librarian’s role in this stage is minimal. The researchers and methodologists refine the coding form and glossary and select the studies that will be included in the project. The studies that meet the criteria for SMITH/META-ANALYSIS 275 meta-analysis will be coded and given the appropriate statistical treatment. If the researcher so chooses, studies that do not meet the exacting standard for meta-analysis can be included in a descriptive statistical analysis. During this time, the librarian should be reviewing and updating the actions taken during data collection. Was a database overlooked? Is there another method to obtain an elusive publication? Is there a librarian or information specialist colleague with a different subject speciality or background that can review the search strategies? Has information in the databases selected been updated so that the searches need to be performed again to retrieve new studies? Has a source been overlooked that might lead to the discovery of an unpublished study appropriate to the project? This is perhaps the biggest gap in the data collection and data evaluation stages. Unpublished studies are by nature elusive. They are often uncovered by checking bibliographies of other studies, word-of-mouth from a colleague or a subject expert, or just plain serendipidity. Of all the roles the librarian plays, tracking down unpublished studies is the most challenging. However, this constant checking and rechecking is imperative to the meta-analysis process. It is up to the librarian to keep the search and retrieval aspects of the project accurate, thorough, and current. Stage four of the meta-analysis process is data analysis and interpretation. During this stage, subject, methodology, and statistical researchers begin the tedious process of reviewing the results of the statistical treatment performed on the studies selected in the data evaluation stage. While the librarian’s interaction with the team is minimal, the librarian can perform a significant function. Because the librarian has distance from the project and a different perspective, the librarian can help to focus discussions and act as a facilitator when the entire team meets. It is also important for the librarian to look ahead to stage five-dissemination of results. If they are to be useful to others, the results of research projects must be reported in the literature. The librarian may not be heavily involved in the actual writing of the research results but can help identify journal titles appropriate for dissemination. Most researchers know the journals in their major subject area and to which of those journals they wish to submit their article. However, given the multidisciplinary nature of many of these studies, the researcher may ask the librarian to assist in identifying majorjournals in other disciplines that might have an interest in this project. To continue with the family violence example, the social work researchers will know in which social work journal they wish to publish. But they might not know the legal, education, or psychological journal possibilities. In addition to identifying journal titles, the librarian might raise the issue of publishing outside the major subject area. The researchers may be so focused on getting published in their subject area that they might 276 LIBRARY TRENDS/FALL 1996 overlook other publishing opportunities. Also, while the results will be reported in the appropriate subject journal, there might be possibilities for reporting unique aspects of the methodology used or describing the process of performing this kind of research in the ancillary subject areas. The librarian also needs to address the aspect of electronic access to the publication. The research team needs to answer the question, “What electronic database would I use to search for articles on this topic?” or “What database(s) would the people that I want to read this article access if they want information on this topic?” Informing the research team of the various databases in which a journal is indexed is a major role for the librarian in this stage of the project. The words interdisciplinaryor multidisciplinary have not been frequently used during the discussion of the librarian’s role in a meta-analysis project. When dealing with a meta-analysis or integrative review of research, interdisciplinary is a given. Looking beyond the composition of the team, the area chosen for research will require an interdisciplinary approach. Even a problem question as innocuous sounding as Does early intervention by a social worker decrease violence directed by a parent toward a child? requires a broad approach. For example, how did the social worker become involved? If the police requested the social worker, then legal databases and informational sources must be searched. Did a teacher request intervention? Then education sources are needed. Did someone in the family seek counseling? Then psychological and psychiatric information is required. The range of possibilities is great and none of them must be overlooked. The search may find nothing useful, but the researcher must document that the appropriate sources were identified and searched and what resulted from this process. The librarian’s unique qualifications in this area are vital to the success of this stage of the research project. As a conclusion to this section, Smith and Stullenbarger (1989) combined from several sources the following list of questions to be used as a suggested guideline for evaluating a meta-analysis or integrative review of a research project. 1. Are the purpose and problem questions specified? 2. Does a theoretical framework serve as the basis for coding, hypothesis testing, and interpretation of results? 3. Are descriptions provided to ensure representativeness of the sample? 4. Are decision rules made explicit at each step of the process? 5. Is there sufficient similarity among constructs, treatments, and control groups for study comparisons? 6. Is the unit of analysis consistent across studies? 7. Are checks for reliability and bias described at each step of the